fix: update WWE scraper to parse HTML instead of defunct API #13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Parse Main Card and Prelims sections from UFC.com event pages - Extract fighter names with proper spacing (first + last name) - Include weight class for each fight (e.g., Bantamweight) - Add broadcast/streaming info (e.g., Watch On Paramount+) - Limit to 5 fights per card section to keep descriptions readable Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
- Scrape events from official allelitewrestling.com website - Parse dates, times, venues with proper timezone handling (PT, MT, CT, ET, AEDT) - Handle Wix JavaScript-rendered page with text-based parsing - Differentiate PPV events (4hr) from regular shows (3hr) - Add comprehensive unit tests (22 tests covering datetime parsing, event fetching, calendar class) - Use proper logging instead of print statements Tested locally with 17 events successfully scraped from AEW website. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
The previous WWE API endpoint (wwe.com/api/events-search-results) now returns 404. Updated the scraper to parse the HTML events page instead. Changes: - Scrape https://www.wwe.com/events/results/all-events/all-dates/0/0/all/all - Parse table rows for event title, date, time, venue, and location - Add timezone detection from city/state (US states + international) - Convert local venue times to UTC - Set event duration based on type (4h for PLEs, 3h for RAW/SmackDown, 2h for NXT) - Handle duplicate events (page sometimes shows same event twice) - Add comprehensive tests (29 test cases) Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
- Add `show` query parameter to filter events by type: - raw: Monday Night RAW events - smackdown: Friday Night SmackDown events - nxt: NXT events - ple: Premium Live Events (WrestleMania, SummerSlam, etc.) - all: All events (default) - Add classify_show() function to categorize events - Add SHOW_KEYWORDS constant for classification rules - Include show_type field in returned event data This enables individual calendars for each WWE show type: - /wwe/events?show=raw - /wwe/events?show=smackdown - /wwe/events?show=nxt - /wwe/events?show=ple Adds 12 new tests for show classification and filtering. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
The previous WWE API endpoint (
wwe.com/api/events-search-results) now returns 404. This PR updates the scraper to parse the HTML events page instead.Changes
https://www.wwe.com/events/results/all-events/all-dates/0/0/all/allTest Plan
🤖 Generated with Claude Code